Jackson's Theorems and the Number of Hidden Units in Neural Networks for Uniform Approximation

It has been known for some years that the uniform-density problem for neural networks has a positive answer: Any real-valued, continuous function on a compact subset of R d can be uniformly approximated by a neural network with one hidden layer. We contribute here to the related complexity problem. We call an activation function : R ! R nearly exponential if, after suitable aane transformations in the source and target spaces R, approximates the exponential function arbitrarily closely on the negative half line. This class of functions contains the functions 1=(1 + e ?t), tanh(t), and e t ^ 1. A neural network is of nearly exponential type if the hidden neurons carry nearly exponential activation functions. We show among other things that the order of approximation of a Lipschitz-continuous function by neural networks of nearly exponential type is O(N ?1=d), N being the number of hidden neurons. We also expose constants in the estimates and derive an algorithm for eective approximation of a continuous function by a neural network of given nearly exponential type. The method of proof consists in approximating the continuous function by exponential sums and these, in turn, by neural networks of nearly exponential type.